Expecting the Unexpected: Warehousing and Analyzing Data from ITS Field Use

One should expect the unexpected when deploying intelligent tutoring systems. This paper describes a case study in collecting, warehousing, and analyzing field usage data from two language and culture learning environments, to understand what happened when they were deployed. A data warehousing system, Hoahu, was used to process the raw data and transform it into a relational database to facilitate queries and analysis. The system also supported data annotation by subject matter experts to facilitate comparison of automated assessments against human raters. Errors and inconsistencies in the data were identified and corrected. The resulting data warehouse has proven valuable for understanding the trajectory of learning over extended periods of time and analyzing the strengths and weaknesses of complex interactive subsystems such as spoken dialog systems.